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ABSTRACT 

The Boeing Company is involved in partnerships with 
Washington state schools in the area of applied academics* Over the 
past 3 years, Boeing offered grants to 57 high schools to implement 
applied mathematics, applied communication, and principles of 
technology courses. Part 1 of this paper gives an overview of applied 
academics by examining what applied academics is and why it is 
important to current educational reform. Part 2 describes the 
multiple approaches the Northwest Regional Educational Laboratory has 
used in evaluating the project and provides a summary of the 3-year 
evaluation results. The aim of applied academics is to make learning 
more meaningful by providing concrete applications of theoretical 
principles within the context of work-relevant examples. The 
evaluation approach used for the Boeing project included case 
studies, surveys, student tests, and test analyses that involved more 
than 126 teachers and 2,544 students. Evaluation to date indicates 
positive responses from teachers and students and suggests a 
notential role for applied academics in educational reform. The 
necessity of multiple approaches to evaluation is amply demonstrated 
in the early stages of this program evaluation. (Contains 18 
references ♦ ) (SLD) 
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A Multiple Approach to Evaluating Applied Academics 



Introduction 

Convinced that applied learning methods contribute to current educational reform and to a 
better-qualified workforce in the future, The Boeing Company in Washington state is 
involved in unprecedented partnerships with schools in the area of applied academics. 
Since 1990, Boeing has invested over three million dollars in grants to Washington high 
schools to implement applied academic programs. Boeing contracted with the Northwest 
Regional Educational Laboratory (NWREL) to conduct a three-year (1990-94) evaluation 
of the project. 

Over the past three years, Boeing offered grants to 57 high schools in Washington state to 
implement Applied Mathematics (AM), Applied Communication (AC), and Principles of 
Technology (PT). Boeing played the following roles in the project: 

1 . Used its influence as the state's major employer to convince education policy 
makers at the local and state levels of the benefits of applied academics 
courses. 

2. Validated new kinds of basic skills that are needed in the workplace 

3. Provided seed money to local schools to purchase AA equipment and materials 
and the reimbursement of release time for training of teachers 

4. Provided opportunities for applied academics teachers to see their subject areas 
being used in state-of-art v/ork settings 

5. Provided incentives and resources to encourage high school and comm' nity 
college faculty to link their programs and to encourage community college 
faculty to use applied approaches ?.n selected curricula 

This paper consists of two parts. Part I gives an overview of applied academics by 
examining what applied academics is and why it is important in current educational 
reform. Part II presents the multiple approach NWREL used in evaluating the Boeing 
Applied Academics Project and a summary of three-year evaluation results. It is the hope 
of the writers that this paper will enhance the understanding of the nature of applied 
academics and that the evaluation approaches described in this paper will be useful for 
those who conduct similar evaluations of applied academics curricula in the future. 

An Overview of Applied Academics 

Marked by the publication of A Nation at Risk in 1983 as its beginning, educational reform 
in the 1980s was largely directed at halting a frightening decline in students 1 basic skills 
performance. While the back-to-basics movement may have improved academic rigor, it 
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did not engage many middle-level or at-risk students who saw more mathematics, more 
science, and more requirements as more reasons to turn off or drop out. At the same time, 
more occupations today are becoming very complex — requiring greater skills, for 
example, in solving problems, understanding systems, and working as team members. 

Most of the educational reform reports over the past ten years focus on strengthening the 
curriculum of college-bound students with an increased emphasis on academic learning. 
However, more than 50 percent of students do not enter college, and job requirements are 
continually rising. These students too need a strong understanding of the basics (Grant 
Foundation, 1988). 

Applied academics is part of a larger effort to make high school curriculum relevant to all 
students, a common goal of educators, business, and community leaders. The aim is to 
make learning more meaningful by providing concrete applications of theoretical principles 
within the context of work-relevant examples- 
Applied Academics — The What 

"Applied academics" is a generic term used to describe curricula developed over the past 
decade that show the work relevance of subjects such as physics, mathematics, and 
language arts. These curricula aim particularly at the two middle quartiles of students who 
may find "general" and "college-bound" classes irrelevant. Applied academics curriculum 
packages have been developed through multi-state consortium effoits by two 
organizations: the Center for Occupational and Research Development (CORD) and the 
Agency for Instruction Technology (AIT). They usually include hands-on laboratory 
activities for students as well as high-interest videos to draw student's attention to the 
real-world applications of the concepts taught. Below are descriptions of three applied 
academics programs supported by The Boeing Company. A fourth consortium-developed 
package is Applied Biology/Chemistry which was not completed in time to be included in 
the Boeing Applied Academics Project. 

Applied Communication. AIT developed the AC course to teach students the 
communication skills needed for success in the workplace. Developed in conjunction with 
state departments, provincial ministries of education, and educators from 42 states in the . 
United States and several provinces in Canada, the learning materials are divided into 1 5 
instructional modules that include 1 50 lessons. They can be used to broaden existing 
courses or as the basis for a year-long course. Each module consists of a series of ten 40- 
to 5 5 -minute lessons that incorporate a variety of learning activities and experiences. 

Lessons 1 through 7 of each module provide instruction and practice in communication 
skills as they are generally used in the workplace. Lessons 8 through 10 feature activities 
designed to develop and refine communication skills in five major occupational areas: 
agriculture, business/marketing, health occupations, home economics, and 
technical/trade/industrial. Each module features two video programs. The student work- 
text supplies individual task sheets with lists of goals and objectives, background 
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information, observation checklists, self-evaluation forms, worksheets, schedules, letters, 
and charts. 

Applied Mathematics. With financial assistance from 42 state vocational education 
agencies and guidance from mathematics and vocational educators, CORD developed 33 
units of AM. The materials are designed to meet the needs of students in the two middle 
quartiles of the high school population. The 25 units'consistently use hands-on activities 
and work-based applications to transform abstract concepts into concrete experiences. In 
the 1992-93 school year, nearly 12,800 sites in 50 states taught AM to an estimated 
390,000 student. 

The overall course includes materials that focus on arithmetic operations, problem-solving 
techniques, estimation skills, measurement skills, geometry, data handling, simple 
statistics, and the use of algebraic formulas to solve problems. Firs, year level materials 
are designed for use in a one- to two-year course for academic credit toward high school 
graduation. Alternatively, they may be used in part or infused as needed into existing 
vocational courses. Written generally at an eighth-grade reading level, the materials are 
deemed appropriate for high school students in grades 9 through 12 who are not 
necessarily baccalaureate bound. Second-year level materials are now available and being 
used by some high schools and community colleges across the nation. 

Principles of Technology. PT has now been implemented in some 7,000 high schools in 
50 states with approximately 185,000 students. In their 1988 review cf materials, the 
American Association for the Advancement of Science cited PT materials as the best 
technical physics curriculum available. Developed by a consortium of 47 states and two 
Canadian provinces, PT is a high school course in applied science aimed particularly at 
vocational-technical students to help meet their needs to better understand more 
technology. Students taking PT may go on to college to complete degrees in areas such as 
industrial engineering; others complete associate degrees at a community college or go 
directly to work. The curriculum covers 14 units of study in a two-year period — though 
many schools choose to offer only the first seven units in a one-year course. The 14 units 
of study are: force, work, rate, resistance, energy, power, force transformation, 
momentum, waves and vibration, energy converters, transducers, radiation, optical 
systems, and time constraints. 

Principles of Technology is being taught nationally by science teachers, by vocational 
teachers, and by teams of both in some schools. Some special training is required, and time 
is needed to make sure a laboratory exercise is set up correctly for each unit. High schools 
may choose to offer students elective credit in science, vocational education, and/or 
mathematics depending on local and state policies. In Washington, staff training is 
provided each summer through the Office of the Superintendent of Public Instruction. 

Common Features. According to CORD, the effectiveness of the materials in each of the 
above applied academics curriculum packages was validated through field testing. All 
these materials contain the follow features: 



• Use modularized student units 

• Incorporate teacher-empowering guides for each unit 

• Include competency-based objectives 

• Are enhanced by an instructional video for each unit 

• Are written at an estimated eighth-grade reading level 

• Target secondary vocational students as the primary audience; also useful in 
postsecondary adult learning settings 

• Emphasize holistic learning 

• Can be infused into vocational courses or taught alone as a credit course by 
either vocational or academic instructors — or a team that includes both 

• Are not meant to replace "traditional" academic courses for the top 25 percent 
of the student population 

• Emphasize developing teamwork skills in students 

Applied academics curricula have also been developed in the humanities by Seattle 
Community College in conjunction with The Boeing Company and in economics by the 
National Headquarters of Junior Achievement Inc., Colorado Springs, Colorado. 

Applied Academics — The Why 

Across the Northwest and the nation, teachers report that many of today's youth (those at 
risk, the "neglected majority," and the gifted alike) fail to see relevance in their school 
studies. Many subjects including science, mathematics, history, economics, and language 
arts are still taught in isolation, with little or no reference to how these subjects are applied 
in the workplace and in other areas of life, or how they fit with other school subjects. This 
is occurring at the same time that jobs are requiring higher proficiency in basic skills and 
technological understanding. 

A recent four-year study conducted by the American Association for the Advancement of 
Science (1993) points out that existing science curricula try to cover too much, do not 
teach enough practical applications of science, and fail to integrate the subject with 
mathematics and technology. The study strongly recommends that students be taught how 
science impacts our lives instead of memorizing many irrelevant and useless details. 

An applied academic curriculum gives more emphasis to concrete applications and 
problems solving skills than to theoretical principles in isolation. It puts the abstract 
concepts of geometry, algebra, and physics into practical, hands-on experiences involving 
real life. The same concepts found in academic disciplines are taught, but they are 
wrapped in real-life tasks. In addition, the approach recognizes the importance of 
teachers' addressing differences in learning styles among students and the need of students 
to learn transferable knowledge and skills to succeed in the workplace. The concept of 
applied academics is rooted in experiential learning and a human being's natural tendency 
to search for meaning (Parnell, 1993). 
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The National Assessment of Vocational Education (NAVE) (Wirt et al, 1989) empirically 
tested the idea that high school vocational education contributes to the development of 
students' academic skills and that some students learn academic skills more readily in an 
applied context. Using data from a nationally representative sample of high school 
students from the class of 1982, they found that: (1) mathematics-related vocational 
courses, as structured in the early 1980s, provided fairly low levels of applications of 
mathematics; (2) mathematics-related vocational courses accounted for approximately 1 8 
percent of all vocational courses for all students; and (3) for college-bound students, 
participation in a specific vocational mathematics course (such as business mathematics) 
increased mathematics proficiency equivalent to participation in an algebra course (pp. 79- 
81). The NAVE study concluded that "an objective of federal policy in vocational 
education should be to encourage the expansion of academic learning in vocational 
education and the integration of academic and vocational curricula" (p. 83). 

In his 1989 testimony before the Senate Subcommittee on Education, Arts, and 
Humanities, Charles Benson, director of the National Center for Research in Vocational 
Education, stated, "The case for integration [of academic and vocational preparation] 
stands on three main — and somewhat interrelated — arguments: economic necessity; 
findings from the field of cognitive science; and social justice with respect to the 
distribution of academic and. vocational learning." The concept of integrating academic 
and vocational education is a major premise of the New Carl Perkins Vocational and 
Applied Technology Education Act of 1990. 

In their report at the second annual staff development conference sponsored by the State 
Vocational Education Consortium of the Southern Regional Educational Board (SREB), 
Bottoms and Korcheck (1990) laid out a meaningful statement on the need for integrating 
academic and vocational education: 

The thinking and problem-solving skills of high school students will develop 
more readily if they understand the connection between what they are leaning 
and how it can be used. One way students can achieve this insight is if 
meaningful applied learning activities are integrated into communications, 
mathematics, and science courses and are coordinated with instruction in 
vocational courses. Once students understand the application of academic 
knowledge, they are far more likely to recall and apply information than if they 
rely on rote memorization (p. 16). 

Applied academics curricula are deliberately designed to incorporate as many facets of 
contextual learning as possible. These facets, according to Dan Hull (1993), include 
relating, transferring, applying, experiencing, and cooperating. The following is a brief 
summary of Hull's definitions and some of the examples he gives for each of these facets. 

Relating. Relating is learning in the context of life experiences. In Principles of 
Technology, for example, the unit on momentum uses the familiar experience of driving a 
car and the common knowledge of what happens in an automobile collision to help 



students relate two basic physics concepts — impulse and change in momentum — to 
everyday life. 

Transferring. Transferring is learning in the context of existing knowledge and builds 
upon what the student already knows. Applied academics courses often encourage 
students to reflect on what they know about a subject before they begin a sequence of 
study. In an Applied Communication module on communicating with clients and 
customers, for example, students examine a series of photographs that show two people, a 
parent and a day-care worker, interacting. Students are asked to identify the feelings and 
attitudes of each person toward the other and to reflect on the importance of nonverbal 
behaviors in the resolution of the exchange. This exercise helps the students become aware 
of their own prior knowledge of nonverbal communication and thus prepares them to read 
a text discussion of nonverbal behavior and to carry out an observation exercise. 

Applying. Applying is using concepts and information in a useful context or projecting 
students into an imagined future( a possible career) and/or into an unfamiliar location (a 
workplace). For example, the Applied Mathematics unit on quality assurance and process 
control begins with a narrative about Corie, a newly hired quality inspector for a tire 
company. The story makes clear that Corie will be using mathematical concepts such as 
standard deviations, mean values, and data spread to carry out significant responsibilities 
within the company. 

Experiencing. Experiencing is learning in the context of exploration, discovery, and 
invention. In applied academics courses, students are allowed to experience activities that 
have a direct relationship to work. The laboratories are often based on actual workplace 
tasks. For instance, an Applied Biology/Chemistry lab addresses the student like this, 
"You are a cattle rancher with a cow in estrus. You have just received your shipment of 
frozen semen from a bull that has qualities you would like to breed into your stock. The 
sperm was shipped to you..." The lab goes on to instruct students to compare the viability 
of actual bull semen samples and to evaluate the effect of handling procedures on sperm 
viability. 

Cooperating. Cooperating is learning in the context of sharing, responding, and 
communicating with others. Cooperating is a primary instructional strategy in applied- 
academics courses. This strategy helps students learn the material and is also consistent 
with the occupational focus of applied academics. In applied academics courses, many 
classroom assignments are designed for students to cooperate with others to complete the 
assignment. 



Educational Themes Related to Applied Academics 

Applied academics is often understood as a mere attempt to integrate academic and 
vocational education or profession/technical education. However, this integration goes 
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beyond a curricular theme and has significant implications for a reconstruction of high 
school itself Grubb et al (1991) outlined the implications of creating applied academics 
courses: 



• [improve] the teaching of all subjects by replacing the teacher-driven, didactic, 
and de-contextualized methods of most academic classrooms with the student- 
centered, project-oriented, and contextualized methods used in the best 
vocational classes 

• [reduce] the isolation of teachers by providing new opportunities and motives 
for collaboration 

• [reduce] tracking and segregation of students by eliminating or at least 
weakening the divisions between vocational and academic students, (pp. vii- 
viii). 

The concept of applied academics overlaps with a number of other important educational- 
themes such as "Experiential learning/learning styles," "Interdisciplinary curriculum," and 
"Workforce preparation." 

Experiential learning/learning styles. Experiential learning emphasizes the cognitive role 
of real life involvement which forms a central part of applied academics. In his discussion 
of the variety of learning styles, learning theorist David Kolb (1984) observes that learners 
perceive information through feeling, doing, watching, and thinking. While all these four 
learning styles are integrated, most students will show a preference either as concrete 
learners (doing/feeling) or abstract learners (watching/thinking). Kolb's study of student 
responses found that only a small percentage of all students have a strong ability to learn 
by thinking and watching and the majority of students tend to perceive and process 
information through some kind of concrete experiences and/or experimentation. 

Interdisciplinary curriculum. Applied academics is breaking down the artificial barriers 
between the academic world and the so-called vocational/technical world. At the National 
Center for Research in Vocational Education, Grubb et al (1991) identified eight models 
for integrating academic with vocational education. Two models make use of applied 
academics. One approach integrates vocational' and academic teachers by using academic 
teachers to teach applied academics courses modified for particular occupational curricula 
or to teach individual lessons for vocational students. A second model makes the academic 
curriculum more vocationally relevant by incorporating vocational applications and 
making academic topics more relevant to all students. Applied academics courses such as 
AM and PT can be academically sound and vocationally important. 

Workforce preparation. One of the contextual facets of applied academics is its 
relevance to the world of work. Many examples included in applied academics courses are 
directly obtained from the workplace. Applied academics is in faci the foundation of 
today's Tech Prep program which seeks to connect education with occupational choices. 
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Without trading students, Tech Prep is a program which links secondary and 
postsecondary education with the particular aim of preparing the workforce of tomorrow. 
(Hull and Parnell, 1991). 

Evaluating Applied Academics 

As applied academics is still comparatively new in most schools, significant data regarding 
the success of the curricula are only now beginning to accrue. Most evaluations of applied 
academics conducted so far focused only on students' performance on certain tests (see, 
for example, McKillip et al 1992, Dugger and Johnson 1992, Dugger 1989, Baker et al. 
1991, and Wicklein, 1991). Although students 7 performance on relevant tests is a good 
indicator of the effectiveness of applied academics, our expectations and assessments 
should also reflect the different approaches to teaching and leaning in applied academics. 
Hull (1993) and his colleagues — especially Leno Pedrotti, the principal designer of several 
applied academics courses, would consider an applied academics curriculum to be a 
success if fhe following criteria apply: 

• Students are able to transfer knowledge from academic content to vocational 
applications and from school to the workplace 

• Students are not afraid to take academic subjects such as mathematics and 
science 

• Student display more interest, motivation, and understanding of the value of the 
subject and of school in general than they did in classes taught by traditional 
methods 

• The applied course is as challenging as the traditional "college-prep" course on 
the same subject — not low level or watered down 

• The student population that has traditionally done poorly in academic subjects 
displays improved performance 

• Applied courses receive the same recognition and acceptance from universities 
and colleges as do the traditional courses with the same content (pp 86-87) 

NWREL was able to incorporate all these criteria into its evaluation design of the Boeing- 
sponsored applied academics project by using a multiple approach that included cases 
studies, surveys, student tests, and test analyses. In addition to using the criteria described 
above, NWREL also examined the implementation process of applied academics in the 
context of other educational reforms taking place in the state of Washington. 

This section presents the different approaches used in evaluating the Boeing Applied 
Academics Project and a summary of evaluation results from 1990-91 to 1992-93. 
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Comprehensive evaluation results were documented in three separate reports (Owens 
1991, Owens and Wang 1992, and Owens and Wang 1993), 

Key Evaluation Questions 

In evaluating the Boeing Applied Academics Project, NWREL looked at the four 
components of the project: project environment, participants, project operation, and 
outcomes. Listed below are key evaluation questions for each component. 

1 . Project Environment 

1.1. What conditions and needs existed at each participating school before the 
beginning of the Boeing funded project? 

1.2. Were the applied academics courses(s) being used before Boeing funding? 

1.3. What was the extent of collaboration between: (a) academic and vocational 
teachers and (b) the high school and community college? 

1 .4. What was the motivation to participate in the Boeing project? 

1.5. How does applied academics fit with other educational reform efforts in 
participating schools? 

2. Participants 

2. 1 . What are characteristics of the students participating in the applied academics 
courses? 

2.2. What are the characteristics of applied academics teachers? 

3. Project Operations 

3.1 To what extent were the applied academics courses used or designed versus 
being locally modified? 

3.2 How many units of each applied academics course were taught? 

3.3 Were the units used as a single course or infused into existing curricula? 

3.4 What efforts occurred, if any, to encourage academic and vocational teachers 
to collaborate? 

3.5 What efforts occurred, if any, to encourage high schools and local 
community colleges to collaborate? 



4. Outcomes 



4. 1 . What percentage of students passed the applied academics course? 

4.2. What gains were made on test scores? 

4.3. What percentage of students in applied academics plan to: (a) enter two-year 
technical or community college and (b) complete a fou/-year degtee? 

4.4. What percentage of applied academics students received: (a) academic credit 
and (b) vocational credit for applied academics courses? 

4.5. What are student reactions toward the applied academics course? Has it 
influenced a change in their career goals? 

4.6. What were the teacher and administrator reactions to the applied academics 
course? 

4.7. What were the teacher reactions *o the Boeing internship experience? 

4.8. How many articulation agreements with colleges were made as a result of 
this project? 

4.9. What were the perceptions of Boeing Company officials regarding the 
project and its benefits? 

Different data collection procedures were used so that the weakness of each procedure 
was reduced by the other. Using the above key evaluation questions as a framework, case 
studies, surveys, and testing collected various data regarding the effectiveness of the 
program. . 



Case Studies 

Case study data supplement test and survey data to help cvaluators gain a better insight 
into how applied academics operate and fit within the total school curriculum. In the 
1990-91 and 1991-92 school years, nine schools sewed as case study sites. Selection 
criteria included: (1) area of the state, (b) size of the community, (c) ethnic diversity of the 
student population, and (d) type of applied academics classes. A wide of variety of schools 
ranging from small rural schools to large comprehensive urban schools were selected. 

The case studies employed: (1) classroom observations to sec how the applied academics 
classes were taught, (2) structured interviews with the instructors and a sample of 
students, and (3) interviews with others involved in the project such as principals, 




counselors, school board members, other academic and vocational teachers, parents, and 
local business people. 

Summary of case study results* The case studies show a variety of positive effects. 
Specific results include: 

• Without funding from Boeing, many schools say that they are unable to offer 
applied academics courses, especially those requiring significant equipment 
purchases, such as PT. 

.• Competent and creative teachers are able to go beyond the concepts in applied 
academics and reach their students with real-world and workplace examples. 

• Successful applied academics program require strong administrative leadership and 
support. 

• Team teaching works well for applied academics courses. Applied academics 
classes are taught by teams in some schools and give students an important model 
for teamwork skills they will need in the workplace. 

• Applied academics teachers need to communicate the intent of the applied 
academics offerings more effectively to sludents, parents, employers, and other 
teachers who are not directly involved with applied academics. 

• The Boeing-funded project has an impact beyond its effect on students in 
classrooms. What Boeing offers to these schools is not only dollars but also 
prestige and the business sector's commitment to applied academics. 

Surveys 

NWREL developed a set of teacher and student surveys consisting of Principles of 
Technology Teacher Survey, Principles of Technology Student Survey, Applied 
Mathematics Teacher Survey, Applied Mathematics Student Survey, and Applied 
Communication Teacher Survey. More than 126 teachers and 2,544 students responded to 
these surveys. 

Student surveys were designed to: 

• Collect some demographic information and identify students' academic 
backgrounds, their educational aspirations and their motivations for taking the 
applied academic course 

• Obtain students' feedback on the content of the applied academics course they 
were taking and the way it was taught compared to other traditional academic 
courses 
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• Identify students' ability to relate ideas from the applied academics course to the 
world of work and their desire for more mathematics and science classes 

Teacher surveys were designed to: 

• Identify academic backgrounds of the teachers' students, their own teaching 
backgrounds including their previous teaching experiences with applied academics, 
and their motivations for teaching applied academics 

• Obtain their feedback on the content of the applied ac?idemics course they were 
teaching and the way it was taught compared to other classes they taught 

• Identify the impact of applied academics on student learning as well as on their 
way of teaching 

• Find out the degree of acceptance of applied academics by colleges or universities 

Summary of survey results. The survey results show that both students and teachers feel 
very positive about applied academics. Most applied academics students are from the 
second or third quartiles of the high school student population. Close to 90 percent of 
students and teachers indicate they would recommend applied academics courses to other 
students or teachers. Most teachers and students indicate that applied academics courses 
provide students with opportunities to learn by doing. They are relevant to the real world 
and help students find meaning in what they learn in school. Applied academic courses 
challenge students in problem solving and involve more student-teacher interactions. 
Academic-vocational teams are considered best for teaching applied academics courses. 



The concerns expressecf by students and teachers often include an inaccurate perception 
about applied academics. The word "applied" is often associated with "watered down," 
low-level courses or remedial courses; as a result, some students find themselves 
unprepared for a challenging ctess. 

Student Tests 

Case studies and teacher and student surveys help reveal the environment in which applied 
academics courses are implemented and the attitudes of students and teachers toward 
these courses. To assess student learning of the course content, proper tests are needed. 
NWREL staff developed a set of test instruments in the first year of the evaluation for 
Applied Mathematics, Principles of Technology, and Applied Communication courses. 

Because most Washington teachers in the project include only one or several units of the 
Applied Communication materials in their existing curriculum and many select different 
Applied Communication units to use in their class, it was inappropriate to develop an 
Applied Communication test. As an alternative. NWREL staff chose to develop a pre- and 
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post-assessment simulation that would measure student performance on broad 
communication skills covered in the Applied Communication curriculum. The simulations 
were administered in first and second year only. 

Pre- and post- tests for Principles of Technology and Applied Mathematics were 
administered in the second and third year of the project. In the third year, the Principles of 
Technology post-test was modified to make it suitable not only for Principles of 
Technology students but also for comparable, regular physics students. 

Pre- and post-simulations for Applied Communication. The simulations measured 
students' skills in group communications and problem solving. Two problem contexts 
were used: an overcrowded lunchroom in one company and a defective toaster returned 
numerous times by customers. In each case, teacher > divided the class into groups of four. 
Each group was given a problem sheet. In addition, they were given one page to record 
group solutions to the problem. Specifically, they were asked to decide how they could 
gather more information about the problem and how other employees could be involved in 
helping solve the problem. On a separate review form, each student listed his or her name 
and the names of the other students. After engaging in the group problem-solving activity, 
students were asked to rate themselves and the other members of their group on six 
communication skills. They responded on a five-point scale ranging from "almost always" 
to "almost never" to the following questions: 

1 . Did the member actively participate in the committee's discussion? 

2. Did the member pay attention to the discussion and stay on the topic? 

3. Did the member offer useful ideas to the discussion? 

4. Was the member considerate of other committee members and their ideas? 

5. Did the member communicate ideas clearly and effectively? 

6. Did the member listen well to others? 

The above scales were based on prior research by Davey and Shively in developing the 
Group Performance Rating Form. This form was then revised by the Connecticut State 
Department of Education and used in the Common Core of Learning Assessment 
Project (Shively, 1991). 

Applied Mathematics pre- and post-tests. These tests were developed by Dr. Dean 
Arrasmith of the NWREL Evaluation and Assessment Program and pilot tested in May 
1991 with 86 students involved in the Boeing project. The test covered units one to eleven 
of the Applied Mathematics curriculum. These units were selected rather than the entire 
15 units since many teachers indicate that they have not covered more than 1 1 units. Test 
items were taken or adopted with permission from the Applied Mathematics Assessment 
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unit tests developed by Dr. Jerry Pepple of National Center for Reset in Vocational 
Education at the University of Illinois site. Each had exactly the same percentage of items 
covering each unit. In selecting items to use from the Illinois tests, items were chosen that 
seemed to have the broadest content and displayed a work relevant context. 

The NWREL Applied Mathematics test results were analyzed by Dean Arrasmith using 
the Item and Test Analysis Program of MicroCat software. The mean score was 18.61, 
and the standard deviation was 6.35. The scores ranged from zero to thirty and the Alpha 
reliability was 80. During the summer, NWREL staff revised the test to eliminate or revise 
poor items including those that had a low-point biserial correlation. 

Principles of Technology pre- and post-tests. These tests were also developed by 
Dr. Dean Arrasmith and pilot tested in May and June of 1991 with 81 students in three 
schools involved in the Boeing project. The tests covered only units one to four of 
Principles of Technology since some teachers reported that they had not covered more 
than that number of units. Test items were taken or adopted with permission from the 
Principles of Technology unit tests developed by the Agency for Instructional Technology 
(AIT) and by Center for Occupational Research and Develepment (CORD). Dr. Jim Shea, 
manager ^f research and evaluation at AIT, provided pre- and post-test item data on these 
items. The NWREL test contained 40 items that were balanced in representation across 
the four Principles of Technology units. The mean score was 24.53, and the standard 
deviation was 7.50. Scores ranged from zero to thirty and the Alpha reliability was 93. 
During the summer, NWREL staff revised the test to eliminate or rewrite poor items 
including that had a low-point biserial correlation. 

To compare students taking applied academics courses to those taking traditional 
mathematics and physics, the Applied Mathematics test was administered in the third year 
of the evaluation to a group of students who did not take the Applied Mathematics course 
and a new physics test was developed by combining Principles of Technology items 
developed by NWREL and items from an introductory physics test designed for the 
American Association of Physics Teachers. Based on point-biserial correlations, items 
from the two tests were selected. These items were then validated by teachers in the field 
and other experts to ensure that they were suitable for both the applied academics student 
groups and the comparison groups. A short version of the student survey was attached to 
each test for students to fill in so that certain variables such as gender, grade level, and 
grade point average could be controlled in analyzing scores from the two groups. 

Summary of student test results* An analysis of variange was run between student 
ratings on the lunchroom simulation and ratings on the toaster simulation. Pre- and post- 
ratings indicate no significant gains in self-ratings in the six communication areas measured 
or in ratings of students by their peers in these same areas. This may be due to the lack of 
integrating sufficient Applied Communication units into existing vocational courses, the 
broad nature of the items being rated, or the unreliability of using students as raters of 
themselves and their peers. 
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A total of 193 Applied Mathematics students took both pre- and post-tests in the first and 
second year of the project. Students score significantly higher on post-tests than they do 
on pre-tests. When Applied Mathematics students' post-test scores in the third year are 
compared with regular mathematics students taking the same test, Applied Mathematics 
students score significantly higher than comparison group students. Applied Mathematics 
female students also do better than their counterparts in the comparison group. 

A total of 787 Principles of Technology students took both pre- and post-tests in the 
second and third year of the project. Test results show that Principles of Technology 
students earn significantly higher scores on post-tests than they do on pre-tests. The third 
year's post-test included items from previous Principles of Technology tests as well as 
items designed for regular physics students. Regular physics students score significantly 
higher on physics items. Female Principles of Technology students score very close to 
their counterparts on both Principles of Technology items and physics items. Principles of 
Technology students who receive mostly As in mathematics, or mostly As in science, or 
have a grade point average of over 3.5 score significantly higher on Principles of 
Technology items but lower on regular physics items than their counterparts in the 
comparison group. 

Student test analysis* Pre- and post-test results of Applied Mathematics and Principles of 
Technology obtained through the second year (1991-92) evaluation of the Boeing Applied 
Academics Project were analyzed by experienced Principles of Technology and Applied 
Mathematics instructors currently teaching in the field. The purpose of the analysis was to 
use the expertise of these field teachers to further interpret the test results and provide 
information to other Principles of Technology and Applied Mathematics teachers so that 
they can improve student performance in weak areas identified from previous test scores. 

The. field instructors analyzed the tests in response to three questions: 

1 . What is covered in pre- and post-tests of Principles of Technology and Applied 
Mathematics? 

2. What are the areas in which students are identified as weak in their test 
performance? 

3. What are your recommendations to Applied Mathematics and Principles of 
TeCiinology teachers to improve student performance in these weak areas? 

Teachers involved in this project responded very positively to the test analysis. They feel 
excited to see the connection of evaluation and improvement of student leaning. They had 
an extensive list of recommendations. Some examples include: (1) teachers need to 
emphasize higher-level thinking skills while providing for basic skills review and 
remediation on an individualized, "need to know" basis; (2) changing students' attitudes 
toward mathematics is essential for real learning to occur (success breeds success!); 
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(3) teaching should be done in teams whenever two or more sections of Applied 
Mathematics are taught; (4) a vocational and mathematics teacher make an ideal match for 
teaching Applied Mathematics curriculum; and (5) there should be a common planning 
period for all Applied Mathematics teachers in the building. Some of the recommendations 
obtained through the analysis were incorporated into Principles of Technology and 
Applied Mathematics classrooms. 

Conclusion 

Applied academics has its theoretical roots in cognitive science and such concepts as 
experiential learning, different learning styles, and multiple intelligences. The key to the 
success of applied academics is "contextual learning." Applied academics fits naturally 
with other educational efforts aiming at connecting schools to the world of work. The 
significance of applied academics courses lies in their approach rather than their content. 
"Applied academics takes classical-education content and then packages it in a new way," 
explains Larry McClure, director of the Education and Work Program at NWREL, "This 
way, students no longer are dealing with theories or make-believe story problems. Instead, 
they are learning through real life activities that are undertaken every day in the U.S. 
workplace. 

Given the nature of applied academics, a single evaluation approach that merely relies on 
students' test performance is not adequate. A multiple approach must be used to measure 
the different indicators of success of applied academics courses. These different 
approaches are complementary so that data collected through one procedure can be 
validated by other procedures. The fact that applied academics programs often interrelate 
with other educational programs requires evaluators to understand the system within 
which applied academics is operating and the effects of any intervening factors at multiple 
levels of the system. This understanding can only be achieved though a multiple evaluation 
approach. 
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